\section{Lexing}

\subsection{Building}
the tags file is as follows

\begin{bashcode} 
$ cat tags
<*_ulex.ml> : camlp4o,use_ulex
<*_ulex.{cmo,byte,native}> : pkg_ulex
\end{bashcode}
%$

Use default myocamlbuild.ml(\ref{myocamlbuild}), and for debugging
process

\begin{bashcode}
  camlp4o pa_ulex.cma -printer OCaml test_ulex.ml -o test_ulex.pp.ml
\end{bashcode}
There's already built-in support for \textit{.pp.ml} in
\textit{ocamlbuild}



\subsection{Implementation}
\label{Implementation}
Its implementation is surprisingly simple. The syntax extension part
\textit{pa\_ulex.cma} is composed of three modules \textit{Cset, Ulex,
  Pa\_ulex}. The Runtime part \textit{ulexing.cma} is composed of two
modules \textit{Utf8, Ulexing}.

\subsubsection{Cset}
\label{Cset}

It defines some util functions for \textit{character set} which was
represented as lists of intervals \textit{(int * int) list}. It
defines some constants and set operations on \textit{character set}.

\subsubsection{Ulexing}
\label{Ulexing}
This module provides runtime support for lexers generated by
\textit{ulex}.  Roughly equivalent to the module Lexing, except that
its lexbuffers handles Unicode code points OCaml type \verb|int| in
the range \verb|0.. 0x10ffff| instead of bytes (OCamltype :
\verb|char|).

You can customize implementation for lex buffers, define a module L
which implements \emph{start,next,mark, and backtrack and the Error
  exception}.  They need not work on a type named lexbuf, you can use
the type name \textit{you want}.

Then, in your \emph{ulex-processed} source, before the first lexer
specification, inset such a line of code \verb|module Ulexing = L|. If
you inspect the processed output by \textit{camlp4}, you can see that
the generated code \emph{introducing Ulexing } very \emph{late} and
actually made use of very limited functions, other functions are just
provided for your convenience at runtime,so you can customize it.

You can refer module \textit{Custom\_lexing} in the source tree for
detailed example.

Refer to
\href{http://www.seas.upenn.edu/~hongboz/hongbo_zhang_files/ulex/Ulexing.html}
{Ulexing API}
for a complete api documentation.

\textit{exception Error} is raised when it can not parse a token from
the lexbuf.

\textit{exception InvalidCodepoint of int} is raised when some code
point is not compatible with a specified encoding.

\subsubsection{Ulex}
\label{Ulex}

This is the main meat of this piece of software, it compiles
NFA to DFA.
\todo{write the principle of the compiling process}

\subsubsection{Pa\_ulex}
\label{sec:Pa_ulex} %label does not need escape...

Provides a syntax extension for convenient use. Here syntax extension
makes sense, since the action needs the whole power of this host
language. If we implement it as a quotation, then we still need the
whole host language parser.

\subsection{Unicode}
\todo{write later, screen, terminal, emacs, easy to confuse...}

\subsection{Usage}    
Ulex does not handle line position, you have only global char
position 


Ulex extends syntax  like this:

\begin{ocamlcode} 
let regexp number = ['0'-'9'] + 
let regexp line = [^'\n']* ('\n' ?)  
let u8l = Ulexing.utf8_lexeme 
let rec lexer1 arg1 arg2 .. = lexer 
   |regexp -> action |..  
and lexer2 arg1 arg2 .. = lexer
   |regexp -> action |...
\end{ocamlcode}

\begin{ocamlcode}
 expr: [
  [ "lexer";
     OPT "|"; l = LIST0 [ r=regexp; "->"; a=expr -> (r,a) ] SEP "|" ->
       gen_definition _loc l ]
 ];
\end{ocamlcode}
\captionof{listing}{Ulexing syntax extension}

\subsubsection{roll back}
\label{roll back}

\textit{Ulexing.rollback lexbuf}, so for string lexing, you can
rollback one char, and \textit{plugin your string lexer}, but
\textit{not generally usefull}, ulex \textit{does not support shortest
  mode yet}. Sometimes the semantics of rolling back is not what you
want like recursive descent parser.

\subsubsection{Combine with other macros}
\label{Combine with other macros}

Since you always need named regex expressions,
use syntax extension macro package \textbf{INCLUDE} to
\textbf{ inline} your code

You can copy some predefined regexps from ocaml source code
\verb| parsing/lexer.ml|.




\subsection{Ulex Example}

Here is the example of simple basic lexer 
\inputminted[fontsize=\scriptsize]{ocaml}{code/lexing/basic_ulex.ml}

Notice that \verb|ocamlnet| provides a fast \verb|Ulexing module|,
probably you can change its interal representation.

\subsubsection{Using other backends}
\label{Using other backends}
\inputminted[fontsize=\scriptsize]{ocaml}{code/lexing/test_net_ulex.ml}

\subsubsection{Attention}

When you use ulex to generate the code, make sure to write the
interface by yourself, the problem is that when you use the default
interface, it will generate \verb|__table__|, and different file may
overlap this name, when you open the module, it will cause a disaster,
so the best to do is \textbf{write your .mli} file.

And when you write lexer, make sure you write the default branch,
check the generated code, otherwise its behavior is weird.


\todo{I have written a helper package to make lexer more available}
\todo{parser-help to coordinate menhir and ulex}

\subsection{Comparison with ocamllex}
\label{Comparison with ocamllex}
\todo{write later}

Ulex support \textbf{unicode}, while ocamllex doesn't, it just scans bytes.

Ulex does not support \verb| as | syntax as ocamllex.
